Method for photographing panoramic image by preventing excessive perpendicular movement with the aid of a displayed icon

ABSTRACT

Disclosed is a method for photographing a panoramic image including the steps of recognizing movement of a corresponding photographing apparatus by comparing a current real-time input image with a previous image through a motion estimation mechanism with exposure compensation, determining a time to photograph each next picture by determining whether movement in a photography direction reaches a preset threshold value, and photographing each next picture by manual or automatic operation at the determined time.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is a Continuation of U.S. Pat. No. 10,178,308 which isa Continuation of U.S. patent application Ser. No. 13/613,608 filed onSep. 13, 2012 and assigned U.S. Pat. No. 9,482,939 issued on Nov. 1,2016 which claims the benefit of the earlier U.S. patent applicationSer. No. 12/082,103 filed on Apr. 7, 2008 and assigned U.S. Pat. No.8,279,264 issued on Oct. 2, 2012 which claims the benefit of the earlierfiling date, pursuant to 35 USC 119 filed with the Korean IntellectualProperty Office on Apr. 12, 2007 and assigned Serial No.10-2007-0036056, the contents of which are incorporated herein byreference.

BACKGROUND 1. Field of the Invention

The present invention relates to digital image photography and moreparticularly to a method for photographing a panoramic image.

2. Description of the Related Art

Generally, a photographing unit of a digital-image photographingapparatus can obtain a picture formed within a focal length of its lens.The obtained picture has a viewing angle range of about 30˜50 degree inthe case of a general camera, which is narrower than the viewing angleof a human being (about 150˜200 degree). Methods for creating an imagewith an angle of view similar to or more than the viewing angle of ahuman require, photographing multiple pictures with an appropriatelychanged photographing angle, and then putting together the photographedpictures in consecutive order to create one image. Such a method isreferred to as panoramic photography.

In a panoramic image photographing mode, a digital-image photographingapparatus photographs multiple pictures in consecutive order in ahorizontal or vertical direction, and stores the pictures in a memory.Later, the pictures stored in the memory are provided to an imageprocessor, and are put together into one image. The image processor maybe internal or external to the photographing apparatus.

In order to eliminate the difference in color and image boundariesbetween the pictures, multiple pictures are photographed in such amanner that the boundaries are appropriately overlapped, and thepictures are aligned in such a manner that the overlapped portions aresynchronized with each other. Then, after the image processingoperations, such as stitching and blending, the multiple pictures aresmoothly put together into one image.

In photographing such a panoramic image, it is important to photographrespective pictures in such a manner that the pictures are aligned asaccurately as possible. Therefore, a user may manually photograph animage using an auxiliary device such as a tripod. In addition, recently,there is provided a method for rotating a photographing apparatus inaccordance with each picture to be photographed in panoramicphotography, by using auxiliary devices. The devices are attached to atripod, etc. and a corresponding photographing apparatus is mountedthereon. An example of such a technology is Korea Patent Application No.2003-0052444, entitled “Camera and Method for Photographing PanoramaPicture” (Application date: Jul. 29, 2003, Applicant: Samsung-Techwin,and Inventor: Bae Sung-chul). Another method a photographing apparatusis provided with a device for detecting a rotation angle, and auser-preset rotation angle of the photographing apparatus. Accordingly,in panoramic photography, when a portable terminal is rotationally movedaccording to the user-set rotation angle, each picture is photographed.

There is further provided a method for more accurately aligningrespective pictures in panoramic photography, without an auxiliarydevice or additional hardware combined with a correspondingphotographing apparatus. In this method, a part of the boundary of apreviously photographed image is displayed while appropriatelyoverlapping an image to be currently photographed so that a user canadjust an appropriate photography position by synchronizing thepreviously photographed image and the currently photographed image witheach other. An example of such a technology is United States PatentPublication No. 2004-0189849, entitled “Panoramic Sequence Guide”(Application date: Mar. 31, 2003, and Inventor: Gregory V. Hofer).

Such panoramic photography requires a user to be more sophisticated andskillful in the operation than for photography of a single-photo, andthus a more convenient operation mechanism and a more efficientphotography mechanism are needed.

SUMMARY

The present invention provides a method for more easily and accuratelyrecognizing the movement of a camera in photographing a panoramic image,so as to use the recognized movement in photographing the panoramicimage.

In accordance with an aspect of the present invention, there is provideda method for photographing a panoramic image, the method including thesteps of recognizing movement of a corresponding photographing apparatusby comparing a current real-time input image with a previous imagethrough a motion estimation mechanism with exposure compensation, inphotographing each picture included in a panoramic image, determiningwhether it is time to photograph each picture by determining whethermovement in a photography direction reaches a preset threshold value,according to the recognized movement of the corresponding photographingapparatus; and photographing each picture by manual or automaticoperation at a moment when the picture is photographed.

BRIEF DESCRIPTION OF THE DRAWINGS

The above and other exemplary features, aspects, and advantages of thepresent invention will be more apparent from the following detaileddescription taken in conjunction with the accompanying drawings, inwhich:

FIG. 1 is a block diagram illustrating a mobile terminal according tothe present invention;

FIG. 2 is a flow diagram illustrating an operation of panoramicphotography according to an embodiment of the present invention;

FIG. 3 is a detailed flow diagram illustrating steps for recognizing themovement of a camera in FIG. 2;

FIG. 4 is an example illustrating an image block with a variable sizefor detecting a motion vector in FIG. 3; and

FIG. 5A and FIG. 5B illustrate integral images.

DETAILED DESCRIPTION

Exemplary embodiments of the present invention are described in detailwith reference to the accompanying drawings. For the purposes of clarityand simplicity, a detailed description of known functions andconfigurations incorporated herein is omitted to avoid making thesubject matter of the present invention unclear.

FIG. 1 is a block diagram illustrating a mobile terminal according tothe present invention. In the present invention, a mobile terminal asshown in FIG. 1, among various apparatuses provided with a digital imagephotography function, will be considered as an example. A mobileterminal provided with a digital image photography function according tothe present invention includes a camera 20, an image processing unit 22,a display unit 24, a control unit 14, a memory unit 16, a key input unit18. The mobile terminal in this illustrated case also includes awireless unit 10, and a wireless-data processing unit 12.

The wireless unit 10 modulates a voice message, an SMS message, andcontrol data of a user to a wireless signal, and transmits the wirelesssignal to a base station (not shown) of a mobile communication network.Also, the wireless unit 10 receives a wireless signal from the basestation, demodulates the wireless signal in to a voice message, an SMSmessage, and control data, and outputs the voice message, the SMSmessage, and the control data. The wireless-data processing unit 12under the control of the control unit 14 decodes voice data receivedfrom the wireless unit 10, outputs the data as audible sounds through aspeaker, turns a user's voice signal input from a microphone into data,and outputs the data to the wireless unit 10. Also, the wireless-dataprocessing unit 12 provides the SMS/control data input from the wirelessunit 10 to the control unit 14.

The camera 20, under the control of the control unit 14, performs thefunction of a general digital camera, and photographs visible lightinput from an outside object. The camera 20 is provided with aphotographing unit 20-2 including a CCD image sensor, etc., and also maybe provided with an illumination sensor 20-4 for measuring illumination,and a distance sensor 20-6 for measuring a focal length from a subject.The image processing unit 22 processes output image data within thecamera 20, and converts the data into digital image data with anappropriate format.

The key input unit 18, by which a user inputs telephone numbers orcharacters, includes keys for inputting number/character information andfunction keys for setting a variety of functions, and outputs such inputsignals to the control unit 14. The display unit 24 may include adisplay component such as a liquid crystal display (LCD), and displaysmessages about various operation states of a corresponding terminal andphotographed digital image data, under the control of the control unit14.

The control unit 14 performs the function of controlling overalloperation of a mobile communication terminal by generally controllingoperations of the respective functional units. In other words, thecontrol unit 14 performs an operation according to signals ofnumber/menu selection input through the key input unit 18, performs anoperation according to an outside photography signal input through thecamera 20, and outputs a camera-photographed image and image signalsrequired for various operations via the display unit 24. When necessary,the control unit 14 accesses content to be output from the memory unit16, or stores content in the memory unit 16. The memory unit 16 storesmany programs and data related to operations of the control unit 14, andalso stores information required for the use of a mobile terminal, andinformation on a photographed image of the camera.

Generally, a mobile terminal with such a configuration performs anoperation related to a general mobile communication service, including acamera function. Besides the above functions, the control unit 14performs may perform panoramic photography operation according to thepresent invention. Also, according to the present invention, the memoryunit 16 stores operation programs and relevant information for apanoramic photography operation in the control unit 14, and whennecessary, outputs such information to the control unit 14.

In panoramic photography according to the present invention, a subjectpicture is input to the mobile terminal in real time, like a movingpicture. When the mobile terminal is moved by a user in a panoramicphotography direction, (or when the mobile terminal is mounted to anadditional device and is automatically rotationally moved by thedevice), the movement of the mobile terminal is recognized by comparinga current input image with a previous image. Then, after determining thedirection and extent of the movement, an operation of obtainingconsecutive pictures is performed in order to create an appropriatepanoramic image. Hereinafter, a panoramic photography operationaccording to the present invention will be described in detail withreference to the accompanying drawings.

FIG. 2 is a flow diagram illustrating an operation of panoramicphotography according to an embodiment of the present invention. Step200 is performed before actual panoramic photography is begun, and isfor setting various photographic environments for panoramic photography.In step 200, the direction (such as, an upward/downward/left/rightdirection) where pictures included a panoramic image are put together isdetermined. Also, the number of photographed pictures to be included increation of a panoramic image is determined.

Such an environment setup operation for panoramic photography in step200 is performed through menu items (usually prepared in a mobileterminal) for setting various functions. For example, when a so-called‘environment setup mode for panoramic photography’ is selected fromamong sub-menus of camera photography menus, a user can input respectiveenvironment settings. A setup and operation program, which are fordisplaying corresponding menu items and recognizing a user's key inputon the key input unit, are previously prepared in the mobile terminal.Also, a part or the whole of such environment items for panoramicphotography, which is set by a user in step 200, may be previouslyprovided as fixed settings appropriately established by a developer.

In step 210, a user selects a so-called ‘panoramic photography mode’from among sub-menus of camera photography menus of the mobile terminal,and starts panoramic photography by photographing the first picture of apanoramic image. The first picture of a panoramic image is stored by ashutter operation of a user in a way similar to usual photography, andthe following pictures are processed by a panoramic photographymechanism according to the present invention. When a panoramic image isphotographed, images input to a photographing apparatus in real-timefrom the start time are stored in the memory unit. Later, such imagesare used for obtaining movement information of the photographingapparatus.

In step 220, according to the present invention, images are input to thephotographing apparatus, like a video image, and the movement of acorresponding photographing apparatus (e.g., a camera) is recognized byusing both a current input image and a previous input image. Inrecognizing the movement of the camera in step 220, a motion vectorbetween matching (most similar) blocks is detected by comparing imageblocks having a appropriately set size with each other within a currentframe and a previous frame. Such a mechanism may be similar to amechanism used for a movement estimation/compensation technology inencoding and decoding of a general video image. In the presentinvention, a motion vector corresponding to a horizontal direction, thatis, x-axis, is defined as m_(i,x), and a motion vector corresponding toa vertical direction, that is, y-axis, is defined as m_(i,y) (herein, iindicates the number of each frame).

In step 230, for panoramic photography in accordance with the movementof the camera as recognized in step 220, the direction where the camerais moved is provided to a user via an appropriate user interface (UI).The direction of the camera movement may be displayed as an appropriateicon, etc. on the display unit.

For example, when it is detected by the motion vector m_(i,x) of x-axisthat the camera currently moves in a left/stop/right direction in acurrent “panoramic photography in progress mode” from left to right, arightward arrow icon may be displayed on a preview screen so as to leadthe movement of the camera in a proper direction for panoramicphotography. Also, in the integration of the motion vector m_(i,y) ofy-axis as in formula (1) below, when the camera exceeds a presetthreshold value (that is, τ₂ in formula (1) below) in an upward anddownward direction, an appropriate type icon for warning of theexcessive movement may be displayed on a preview screen.|Σm _(i,y)|≤τ₂  (1)

Then, in step 240, it is determined whether it is time to photographeach current (next) picture according to the movement of the camera asrecognized in step 220. When it is time to photograph each current/nextpicture, the process proceeds to step 250, or otherwise, the processproceeds back to step 220, and the above steps are repeated.

For example, in the integration of the motion vector m_(i,x) of x-axisas in formula (2) below in “current panoramic photography in progressmode” from left to right, when the integrated value reaches a presetthreshold value (that is, τ₁ in formula (2) below), it is determinedthat it is time to photograph a current/next picture.τm _(i,x)≤τ₁  (2)

In step 250, a corresponding picture is then photographed. The time forphotographing a current picture may be provided to a user via anappropriate user interface (UI). Also, preset settings, for example,manual/automatic panoramic photography, allows the user to perform thephotography operation by shutter, or otherwise. Automatic photographyoperation is performed. The settings for the above manual/automaticpanoramic photography may be set when an environment setup operation forpanoramic photography is performed in step 200.

In step 260, it is determined whether the number of picturesphotographed reaches the total number preset in step 200. When thenumber is less than the total number, the process proceeds back to step220, and the above steps are repeated. On the other hand, when thenumber has reached the total number, the panoramic photography processis completed.

When the process proceeds back to step 220, the values of the motionvectors, especially, the vector sum of the motion vector m_(i,x) ofx-axis (that is, Σm_(i,x)), are reset as an initialized value (0). Also,after the panoramic photography is completed, respective photographedpictures are put together so as to create one panoramic image. In orderto smoothly put together multiple pictures, general image processingmechanisms, including aligning, stitching, and blending, may be applied.

Referring to FIG. 2, when photographing a panoramic image, a user isrequired to move a photographing apparatus at regular intervals in aphotography direction (for example, from left to right). In the presentinvention, it is assumed that image frames are input at substantiallyconstant speed via such a mechanism, and every frame has no motion blureffect. Thus, pictures with a fixed width, which are automaticallyoverlapped, are obtained. Accordingly, in the integration of motionvectors (that is, the motion vector m_(i,x) of x-axis) of input imagesaccording to a photography direction (for example, from left to right),whenever the integrated value reaches a preset threshold value (τ₁), anext picture is photographed. When the photography for a picture is setas an automatic mode, each picture included in a panoramic image isautomatically photographed at an appropriate position (according to thethreshold value τ₁) while a user appropriately moves the photographingapparatus.

In recognizing the movement of the photographing apparatus in step 220using a current input image and a previous input image, a motion vectorbetween image blocks of frames is detected via a mechanism according tothe principles of the present invention. Hereinafter, such a mechanismwill be described in detail.

In the mechanism of matching image blocks of frames and detecting amotion vector, a motion estimation mechanism may be applied. Motionestimation mechanism is usually used for a video image (for example,broadcast images) compression technology, such as MPEG-4, H.264, etc.However, in actual panoramic photography, a preview image of a camera isused. Therefore, differently from broadcast images, exposurecompensation is not properly done, and the correlation between previewframes is not strong. Accordingly, when a general motion estimationmechanism is applied, it is difficult to find an accurate matchingblock. Furthermore, the general motion estimation mechanism is atechnology based on a compression mechanism, and requires a verycomplicated computation mechanism, such as motion vector detection withsubdivided block (macroblock) sizes. Therefore, in consideration ofcharacteristics of the panoramic photography, an appropriate motionvector estimation mechanism is required.

According to the present invention, movement from a previous frame to acurrent frame of input images is recognized by using an image block of acertain area, rather than using a general motion estimation mechanism.Thus, a motion estimation mechanism with exposure compensation isprovided in consideration of principles of panoramic photography. Inphotography, a motion vector has to be quickly computed in such a mannerthat preview operation is normally performed during the computation. Inorder to reduce computational complexity, an integral image may be used.Also, a base block for finding a motion vector is variably adjustedaccording to a possible preview rate, and a relative size and distanceof a photographed object. Herein, due to characteristics of a previewimage, the correlation of frames is not stronger than a compressionimage. In order to maximize the correlation, it is possible to use asingle-block search mechanism. In motion estimation of frames, insteadof using a plane image, it is possible to perform the motion estimationon an image previously projected on a cylindrical panoramic projectionspace in consideration of characteristics of panoramic photography.

FIG. 3 is a detailed flow diagram illustrating steps for recognizing themovement of a camera. The operation of motion estimation according tothe present invention will be described in detail with reference to FIG.3. In step 222, in order to obtain a motion vector between a currentframe and a previous frame, a variable size block is set.

In motion estimation on a preview image, especially, in the case of amobile camera with a low-sensitivity sensor and hand-tremor, when themotion estimation is performed on a small size block (usually, a squareblock in a range of 4-16 pixels) in the manner of a video imagecompression mechanism, it is difficult to find an accurate motion vectorvalue due to accumulated vector errors between blocks. Accordingly, inan embodiment of the present invention, the motion vector is detected ona few blocks (for example, just one block) with a relatively largersize.

The size of the one block is variably adjusted by using the performanceof a preview tool, the size and distance of a photographed object, anddistance information on the area to be searched. FIG. 4 is an exampleillustrating a variably set size of a block within a frame. SR (SearchRange) indicates the area to be searched, and Bk (Block size) indicatesthe size of the block.

A search range SR is an area for searching a block matching a base blockwithin a frame to be searched. It is possible to set the entire area ofthe frame as the search range. On the other hand, instead of utilizingthe entire image of the frame, it is possible to set an appropriate areaas the search range. The search area may be determined based on theposition of a base block within a frame to be searched. In this manner,it is possible to increase the efficiency by preventing unnecessaryportions (such as the edges of a frame) from being searched.

In an embodiment of the present invention, it is possible to variablyset such a search range and a block size. For example, whenphotographing an object at a short distance, or the object size isrelatively large, a more accurate motion vector is obtained byincreasing the block size, and decreasing the search range. On the otherhand, when photographing an object at a long distance, or the objectsize is relatively small, a more accurate motion vector is obtained bydecreasing the block size, and widening the search range.

In the case of a photographing apparatus with a low performance previewfunction, when the search range is excessively widened, it takes a lotof time to find a motion vector due to increased computationalcomplexity, thereby decreasing a frame rate. Also, even if an accuratevector is obtained, the decreased frame rate may cause images betweenframes to be largely different from each other. As a result, errorscauses by the difference are accumulated. Therefore, in extracting amotion vector from a camera preview, it is possible to obtain basicinformation on a photographed object by using object detector, automaticinput information such as a focal length, etc., or user-manual inputinformation, and to set the search range in accordance with a distanceand a size of the object. The user-manual input information may includea distance and an object size, and may also be estimated based onuser-set information including a short-distance mode, a scenery mode,and a portrait mode, etc. In addition, according to preview performanceof a corresponding photographing apparatus, it is possible to set abasic block size in consideration of computational complexity.

The setup of a block with a variable size in step 222 is performed onlyonce when the size of a current block is not set, that is, the firsttime when the extraction of a motion vector is performed. Then, untilthe panoramic photography is completed, a block with the first set sizeis used.

In step 224, a matching block between a current frame and a previousframe is detected by using a block with a size set in step 222.Basically, in detecting the matching block, error information ofrespective compared pixels between blocks is utilized. When the errorbetween two blocks is the least (minimized), the corresponding block isconsidered a matching block.

Usually, for error information for detecting a matching block, an errorfunction such as SAD (Sum of Absolute Difference) or SSD (Sum of SquaredDifference) is used. SAD and SSD generally used in a video imagecompression technology are defined by formula (3) below.

$\begin{matrix}{{SAD}{{E\left( {m,n} \right)} = {\sum\limits_{x,{y \in {block}}}\;{{{I_{1}\left( {x,y} \right)} - {I_{2}\left( {{x + m},{y + n}} \right)}}}}}{SSD}{{E\left( {m,n} \right)} = {\sum\limits_{x,{y \in {block}}}\;\left\{ {{I_{1}\left( {x,y} \right)} - {I_{2}\left( {{x + m},{y + n}} \right)}} \right\}^{2}}}} & (3)\end{matrix}$

In formula (3), Ī₁ and Ī₂ indicate pixel values (luminance or color) ofcorresponding positions on blocks within respective frames (that is, aprevious frame and a current frame). Herein, x and y indicate a pixelposition on a base block in a previous frame, and x+m and y+n indicate apixel position on a searched block in a current frame.

Such functions can be appropriately applied when adjacent two frameshave no difference in exposure, for example, in the case of a videoimage compression environment. However, when adjacent two frames have adifference in exposure, for example, in the case of a image framedirectly input from a subject in an actual photography environment, itis difficult to use the functions as accurate information. Therefore,the present invention uses error functions reflecting the exposuredifference as defined by formula (4) below.

Modified SAD

$\begin{matrix}{{{E^{\prime}\left( {m,n} \right)} = {\sum\limits_{x,{y \in {block}}}{{{I_{1}\left( {x,y} \right)} - {\overset{\_}{I}}_{1} - \left( {{I_{2}\left( {{x + m},{y + n}} \right)} - {{\overset{\_}{I}}_{2}\left( {m,n} \right)}} \right)}}}}\mspace{20mu}{{modified}\mspace{14mu}{SSD}}{{E\left( {m,n} \right)} = {\sum\limits_{x,{y \in {block}}}\left\{ {{I_{1}\left( {x,y} \right)} - {\overset{\_}{I}}_{1} - \left( {{I_{2}\left( {{x + m},{y + n}} \right)} - {{\overset{\_}{I}}_{2}\left( {m,n} \right)}} \right)} \right\}^{2}}}} & (4)\end{matrix}$

Herein, Ī₁ and Ī₂ indicate average values of pixels corresponding toblocks within respective frames (that is, a previous frame and a currentframe). In formula (4), exposure compensation is performed bycompensating the pixel values of each frame block in accordance with theaverage of the image block (in the case of formula (4), the averagevalue is subtracted).

Since the position of a block in the previous frame is fixed (that is, abase block is known), there is no need to obtain Ī₁ more than once.However, in the following frames, in order to find the best matchingblock, through the comparison with the current frame and the previousframe, it is required to search every block with the same size. Thus,whenever the search is performed, it is necessary to obtain the averagevalue of pixels within a corresponding block. In other words, Ī₂(m,n)has to be obtained with regard to every block.

Herein, whenever the total sum and average value is obtained with regardto pixels of every block, computational complexity is increased.Accordingly, the present invention provides a mechanism for decreasingthe computational complexity by using an integral image. In the integralimage, a pixel value corresponding to coordinates indicates the totalsum of pixels extending from the upper left side of the image to thecoordinate. In other words, in FIG. 5A a pixel value of (x, y) indicatesthe total sum of pixels corresponding a dotted block area. Therefore,I_(int g) (0, 0) indicates a pixel value at coordinates (0, 0), andI_(int g) (width−1, height−1) indicates the total sum of all pixels of agiven image.

Through such an integral image, it is possible to obtain the total sumof pixels within a required area by using summation and subtraction ofpixel values corresponding to four vertexes of the area. A total sum ofpixels corresponding to a dotted block area in FIG. 5B can be obtainedby using pixel values of four vertexes (x₁, y₁), (x₂, y₂), (x₃, y₃), and(x₄, y₄) through formula (5) defined below.I _(int g)(x ₄ ,y ₄)+I _(int g)(x ₁ ,y ₁)−I _(int g)(x ₂ ,y ₂)−I_(int g)(x ₃ ,y ₃)  (5)

According to an embodiment of the present invention, an integral imagewith regard to a current frame is previously obtained as describedabove, and then, the average of pixels with regard to every block, thatis, Ī₂(m, n), can be simply obtained by using such an integral image.

After the detection of a matching block between a current frame and aprevious frame in step 224, a motion vector between two blocks (that is,a motion vector m_(i,x) of x-axis, and a motion vector m_(i,y) ofy-axis) is obtained in step 226.

A method for photographing a panoramic image according to an embodimentof the present invention is performed as described above. Although aspecific embodiment has been described, it will be obvious that variousmodifications and variations can be made within the scope of thisinvention.

Usually, a motion vector is detected with regard to a plane image. Suchdetection is efficient only for general compression images and broadcastimages. On the other hand, in the case of panoramic photography,information on a motion vector is required in order to obtain a moreaccurate panoramic image. In another embodiment of the presentinvention, after panoramic photography, each image may be projectedagain on a mosaic plane or a curved plane because consecutive picturesobtained for a panoramic image make a cylindrical form as a camerarotates in a circle. Accordingly, in detection of a motion vector, inconsideration of image movement in a curved form, not a plane form, aninput image is projected on an appropriate projection plane (forexample, a cylindrical plane), and then, the motion vector is detectedwith regard to the projected image. As a result, it is possible toobtain a more accurate value.

Also, a motion vector is determining a preview image for which apanoramic image is input in real time, and whether to photograph animage is determined by using information on the motion vector. Inconsideration of such a characteristic, a motion vector may bedetermined by skipping some frames according to a preview rate of anphotographing apparatus. Usually, in a case of video image compression,since vector information of each frame is important, a motion estimationoperation has to be performed per frame. However, in a motion estimationoperation for panoramic photography according to the present invention,the information of each frame is less important. Accordingly, in anotherembodiment of the present invention, a motion estimation operation maybe performed by appropriately skipping some frames (for example, 1˜2frames) according to related environments. In such a case, computationaltime and complexity may be reduced, and thus, a wider search range issecured. As a result, it is possible to calculate a more accurate motionvector.

The above-described methods according to the present invention can berealized in hardware or via the execution of software or computer codethat can be stored in a recording medium such as a CD ROM, an RAM, afloppy disk, a hard disk, or a magneto-optical disk or downloaded over anetwork, so that the methods described herein can be rendered in suchsoftware using a general purpose computer, or a special processor or inprogrammable or dedicated hardware, such as an ASIC or FPGA. As would beunderstood in the art, the computer, the processor or the programmablehardware include memory components, e.g., RAM, ROM, Flash, etc. that maystore or receive software or computer code that when accessed andexecuted by the computer, processor or hardware implement the processingmethods described herein.

In addition, although in panoramic photography, a photographingapparatus according to the present invention is moved by a user asdescribed above, the principles of the invention can be applied when thephotographing apparatus according to the present invention is mounted toan additional moving device, and is automatically moved. Also, althoughin the above description, panoramic photography is in a horizontal or avertical direction, the principles of the invention can be applied whenthe panoramic photography is in a horizontal-vertical mixed direction(that is, multiple pictures included in a panoramic image are disposedwith a horizontal/vertical mosaic style). Besides, the above various UIsmay be expressed as various ways different from the above description.

As described above, in panoramic photography according to the presentinvention, movement information on a terminal mounted with a camera isrecognized through an image processing operation. Then, it isautomatically determined whether it is time to photograph an image.Accordingly, without an auxiliary device or additional hardware, a usercan easily and accurately photograph an image.

Especially, in motion vector detection according to the presentinvention, a motion estimation mechanism in accordance with thecharacteristics of a preview image of a camera is used. The size of abase block for detecting a motion vector is appropriately and variablyset, and thus it is possible to more simply and accurately extract amotion vector.

While the invention has been shown and described with reference tocertain exemplary embodiments thereof, it will be understood by thoseskilled in the art that various changes in form and details may be madetherein without departing from the spirit and scope of the invention asdefined by the appended claims.

What is claimed is:
 1. An electronic device comprising: an image sensor;a processor; a display; and a memory storing a plurality ofinstructions, wherein the plurality of instructions executed by theprocessor cause the electronic device to perform a plurality ofoperations comprising: in response to receiving an input, startingcapturing of a horizontal panoramic image; displaying, on a previewscreen of the display, an icon for indicating a capturing direction;determining an amount of movement of the electronic device in a verticaldirection during capturing of the horizontal panoramic image; when theamount of movement in the vertical direction satisfied a thresholdcondition, displaying, on the preview screen of the display, a visualindication relating to an excessive vertical movement, during capturingof the horizontal panoramic image; and completing capturing of thehorizontal panoramic image.
 2. The electronic device of claim 1, whereinthe amount of movement of the electronic device is determined based on afirst image and a second image obtained from the image sensor.
 3. Theelectronic device of claim 2, wherein the amount of movement of theelectronic device is determined by comparing the first image and thesecond image obtained from the image sensor.
 4. The electronic device ofclaim 1, wherein the capturing of the horizontal panoramic image iscompleted based on a first image and a second image obtained from theimage sensor.
 5. The electronic device of claim 1, wherein the icon forindicating the capturing direction is displayed when a movementdirection of the electronic device is detected.
 6. The electronic deviceof claim 1, wherein the icon for indicating the capturing direction isat least a shape of arrow direct to the capturing direction.
 7. Anelectronic device comprising: an image sensor; a processor; a display;and a memory storing a plurality of instructions, wherein the pluralityof instructions executed by the processor cause the electronic deviceto: in response to receiving an input, start capturing of a panoramicimage in a direction selected from a horizontal direction and a verticaldirection; display, on a preview screen of the display, an icon forindicating the selected capturing direction; determine movementinformation for the electronic device during capturing of the panoramicimage; when the movement information for the electronic device satisfiesa threshold condition, display on the preview screen of the display avisual indication relating to an excessive movement, wherein: when theselected direction is the horizontal direction, determining the movementinformation for the electronic device comprises determining an amount ofmovement of the electronic device in the vertical direction, and theexcessive movement is an excessive vertical movement; and when theselected direction is the vertical direction, determining the movementinformation for the electronic device comprises determining an amount ofmovement of the electronic device in the horizontal direction, and theexcessive movement is a horizontal movement; and complete capturing ofthe panoramic image based at least in part on a first image and a secondimage obtained by the image sensor.
 8. The electronic device of claim 7,wherein the amount of movement of the electronic device is determinedbased on thea first image and thea second image obtained from the imagesensor.
 9. The electronic device of claim 8, wherein the amount ofmovement of the electronic device is determined by comparing the firstimage and the second image obtained from the image sensor.
 10. Theelectronic device of claim 7, wherein the capturing of the panoramicimage is completed based on the first image and the second imageobtained from the image sensor.
 11. The electronic device of claim 7,wherein the icon for indicating the selected capturing direction isdisplayed when a movement direction of the electronic device isdetected.
 12. The electronic device of claim 7, wherein the icon forindicating the selected capturing direction is at least a shape of arrowdirect to the capturing direction.
 13. A method for generating at leastone image with an electronic device comprising an image sensor and adisplay, the method comprising: in response to receiving an input,starting capturing of a horizontal panoramic image; displaying, on apreview screen of the display, an icon for indicating a capturingdirection; determining an amount of movement of the electronic device ina vertical direction during capturing of the horizontal panoramic image;when the amount of movement in the vertical direction satisfied athreshold condition, displaying, on the preview screen of the display, avisual indication relating to an excessive vertical movement, duringcapturing of the horizontal panoramic image; and completing capturing ofthe horizontal panoramic image.
 14. The method of claim 13, wherein theamount of movement of the electronic device is determined based on afirst image and a second image obtained from the image sensor.
 15. Themethod of claim 14, wherein the amount of movement of the electronicdevice is determined by comparing the first image and the second imageobtained from the image sensor.
 16. The method of claim 13, wherein thecapturing of the horizontal panoramic image is completed based on afirst image and a second image obtained from the image sensor.
 17. Themethod of claim 13 wherein the icon for indicating the capturingdirection is displayed when a movement direction of the electronicdevice is detected.
 18. The method of claim 13, wherein the icon forindicating the capturing direction is at least a shape of arrow directto the capturing direction.
 19. A method for generating at least oneimage with an electronic device comprising an image sensor and adisplay, the method comprising: in response to receiving an input, startcapturing of a panoramic image in a direction selected from a horizontaldirection and a vertical direction; display, on a preview screen of thedisplay, an icon for indicating the selected capturing direction;determine movement information for the electronic device duringcapturing of the panoramic image; when the movement information for theelectronic device satisfies a threshold condition, display on thepreview screen of the display a visual indication relating to anexcessive movement, wherein: when the selected direction is thehorizontal direction, determining the movement information for theelectronic device comprises determining an amount of movement of theelectronic device in the vertical direction, and the excessive movementis an excessive vertical movement; and when the selected direction isthe vertical direction, determining the movement information for theelectronic device comprises determining an amount of movement of theelectronic device in the horizontal direction, and the excessivemovement is a horizontal movement; and complete capturing of thepanoramic image based at least in part on a first image and a secondimage obtained by the image sensor.
 20. The method of claim 19, whereinthe amount of movement of the electronic device is determined based onthe first image and the second image obtained from the image sensor. 21.The method of claim 20, wherein the amount of movement of the electronicdevice is determined by comparing the first image and the second imageobtained from the image sensor.
 22. The method of claim 19, wherein thecapturing of the panoramic image is completed based on the first imageand the second image obtained from the image sensor.
 23. The method ofclaim 19, wherein the icon for indicating the selected capturingdirection is displayed when a movement direction of the electronicdevice is detected.
 24. The method of claim 19, wherein the icon forindicating the selected capturing direction is at least a shape of arrowdirect to the capturing direction.